Environment interactive system providing augmented reality for in-vehicle infotainment and entertainment

ABSTRACT

An environment interactive system includes: a first camera capturing first images of an area external to the vehicle; a second camera capturing second images of an area interior to the vehicle; a telematics control module determining a geographical location of the vehicle; a viewing device displaying items adjacent to, on or in front of detected objects and in a FOV of an occupant. The items correspond to the detected objects; and an infotainment module. The infotainment module: detects the objects based on the first images and the location of the vehicle; based on the second images, tracks at least one of a location of an eye of the occupant or a location of a body part of the occupant other than the eye; and based on the location of the eye and the location of the body part, displays the items via the viewing device in the FOV.

INTRODUCTION

The information provided in this section is for the purpose of generallypresenting the context of the disclosure. Work of the presently namedinventors, to the extent it is described in this section, as well asaspects of the description that may not otherwise qualify as prior artat the time of filing, are neither expressly nor impliedly admitted asprior art against the present disclosure.

The present disclosure relates to in-vehicle infotainment andentertainment.

Vehicles include one or more torque producing devices, such as aninternal combustion engine and/or an electric motor, and are typicallydesigned to be driven for extended periods of time. The vehicles can benon-autonomous or can be partially or fully autonomous. An occupant of avehicle rides within a cabin (or passenger compartment) of the vehicle.Occupants within a vehicle can include a driver and one or morepassengers. If the vehicle is fully autonomous, then all of theoccupants may be referred to as passengers.

Vehicles may include sensors for sensing vehicle surroundings. As anexample, a camera may be used to capture images of an environmentoutside of a vehicle. A vehicle may include forward facing, rear facing,and side facing cameras. Other example sensors for sensing vehiclesurroundings include radar, sonar and light detection and ranging(LIDAR) sensors.

SUMMARY

An environment interactive system for a vehicle is provided. Theenvironment interactive system includes: a first camera configured tocapture first images of an area external to the vehicle; a second cameraconfigured to capture second images of an area interior to the vehicle;a telematics control module configured to determine a geographicallocation of the vehicle; a viewing device configured to display items atleast one of adjacent to, on or in front of detected objects and in afield of view of an occupant, where the items correspond to the detectedobjects; and an infotainment module. The infotainment module isconfigured to: detect the objects based on the first images and thelocation of the vehicle; based on the second images, track at least oneof a location of an eye of the occupant or a location of a body part ofthe occupant other than the eye; and based on the location of the eyeand the location of the body part, display the items via the viewingdevice in the field of view of the occupant.

In other features, the environment interactive system further includes athird camera configured to capture third images of the interior of thevehicle. The infotainment module is configured to, based on the secondimages and the third images, track the at least one of the location ofthe eye of the occupant or the body part of the occupant.

In other features, the body part is a finger of the occupant.

In other features, the infotainment module is configured to track aniris of the eye of the occupant, and based on the location of the irisand the location of the body part, display the items via the viewingdevice in the field of view of the occupant.

In other features, the infotainment module is configured to, based on atleast one of the location of the eye or the location of the body part,display the items in front of the objects and in the field of view ofthe occupant.

In other features, the infotainment module is configured to: receiveinformation and preferences of the occupant; and based on theinformation and preferences, display information about certain ones ofthe objects in the field of view of the occupant.

In other features, the infotainment module is configured to: receiveinformation and preferences of the occupant; and based on theinformation and preferences, display targeted advertisements related tocertain ones of the objects in the field of view of the occupant.

In other features, the infotainment module is configured to: receiveinformation and preferences of the occupant; and based on theinformation and preferences, display targeted coupons related to certainones of the objects in the field of view of the occupant.

In other features, the infotainment module is configured to: receiveinformation and preferences of the occupant; display an item for theoccupant to select corresponding to starting a game; and based on thelocation of the eye and the location of the body part, start the game.

In other features, the game includes displaying questions in the fieldof view of the occupant for the occupant to read and answer.

In other features, the infotainment module is configured to: determineif the vehicle has stopped at a recommended location of a business; anddisplay in the field of view of the occupant a survey question for theoccupant to answer pertaining to the business.

In other features, the infotainment module is configured to: detect whenthe occupant selects one of the objects by gesturing at the object; anddisplay information pertaining to the selected object.

In other features, the infotainment module is configured to: displayinformation on a head up display pertaining to one of the objects; andbased on the at least one of the location of the eye or the location ofthe body part, detect when the occupant selects the one of the objects.

In other features, the infotainment module is configured to: interpret agesture made by the occupant with regards to at least one of one of theobjects or one of the items; and perform an action in response to thegesture.

In other features, the infotainment module is configured to interpret agesture or voice response provided by the occupant to confirm whethereye tracking of a gaze point of the occupant is accurate.

In other features, an environment interactive method for an occupant ofa vehicle is provided. The environment interactive method includes:capturing first images of an area external to the vehicle; capturingsecond images of an area interior to the vehicle; determining ageographical location of the vehicle; displaying items between detectedobjects and the occupant and in a field of view of the occupant, wherethe items correspond to the detected objects; detecting the objectsbased on the first images and the location of the vehicle; based on thesecond images, tracking a location of an eye of the occupant and alocation of a body part of the occupant other than the eye; and based onthe location of the eye and the location of the body part, displayingthe items via the viewing device in the field of view of the occupant.

In other features, the method further includes: tracking an iris of theeye of the occupant; and based on the location of the iris and thelocation of the body part, displaying the items via the viewing devicein the field of view of the occupant.

In other features, the method further includes, based on the location ofthe eye and the location of the body part, displaying the items in frontof the objects and in the field of view of the occupant, on a screen, oron a head up display.

In other features, the method further includes: receiving informationand preferences of the occupant; and based on the information andpreferences, displaying information about certain ones of the objects inthe field of view of the occupant, displaying targeted advertisementsrelated to certain ones of the objects in the field of view of theoccupant, and displaying targeted coupons related to certain ones of theobjects in the field of view of the occupant.

In other features, the method further includes detecting when theoccupant selects one of the objects by gesturing at the object; anddisplaying information pertaining to the selected object.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description, the claims and the drawings. Thedetailed description and specific examples are intended for purposes ofillustration only and are not intended to limit the scope of thedisclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from thedetailed description and the accompanying drawings, wherein:

FIG. 1 is a functional block diagram of an example vehicle entertainmentaccess system in accordance with the present disclosure;

FIG. 2 is a functional block diagram of a vehicle incorporating anexample vehicle system including an example environment interactivesystem in accordance with the present disclosure;

FIG. 3 is a functional block diagram of a portion of the vehicle systemand a portion of the environment interactive system of FIG. 2 ;

FIG. 4 is a perspective view of an interior of a vehicle and a portionof an environment forward of the vehicle implementing the environmentinteractive system in accordance with the present disclosure;

FIGS. 5A-C illustrates an entertainment interactive method in accordancewith the present disclosure;

FIG. 6 is an example view illustrating display of interactive items infront of objects in a field of view of an occupant in accordance withthe present disclosure;

FIG. 7 is an example view illustrating display of an interactiveadvertisement and a corresponding discount in front of objects in afield of view of an occupant in accordance with the present disclosure;

FIG. 8 illustrates an environment interactive method including eyeposition and finger position tracking and displaying of object relatedinformation in accordance with the present disclosure;

FIG. 9 illustrates a personalized environment interactive methodincluding multi-camera eye position and finger position tracking inaccordance with the present disclosure;

FIG. 10 illustrates another personalized environment interactive methodincluding multi-camera eye position and finger position tracking fordisplay of targeted marketing information in accordance with the presentdisclosure;

FIG. 11 illustrates another personalized environment interactive methodincluding one or more cameras for monitoring a gaze point and fingerposition for display of targeted marketing information in accordancewith the present disclosure;

FIG. 12 is an image of an interior of a vehicle illustrating eye andfinger tracking and occupant selection of an object in a field of viewin accordance with the present disclosure;

FIG. 13 is an image of an interior of a vehicle illustrating display ofa text message pertaining to an object in a field of view of an occupantin accordance with the present disclosure;

FIG. 14 is an image illustrating an occupant gesture in response to aquestion provided in a field of view of the occupant; and

FIG. 15 is a view illustrating eye and finger tracking and selection ofan item displayed in correspondence with objects in a field of view ofan occupant.

In the drawings, reference numbers may be reused to identify similarand/or identical elements.

DETAILED DESCRIPTION

A vehicle may include an infotainment system for providing entertainmentwithin a vehicle to entertain occupants of the vehicle. The infotainmentsystem may include a center console display, one or more displays forrear seat passengers, and an audio system. Although these displays andaudio system can provide some entertainment, such as playing musicand/or videos, passengers when traveling for extended periods of timecan become bored. This may occur while the vehicle is cruising, whilethe vehicle is in a congested area, when the vehicle is stopped andrecharging, and/or during other conditions.

The examples set forth herein are directed to providing in-vehicleentertainment and include environment interactive systems allowingoccupants to interact with a view of an environment. The interaction iswith viewed objects and displayed items, such as icons, advertisements,text messages, website pages, applications, and/or other items, whichmay be displayed in the view of the environment and/or in images of theenvironment. The items may be displayed on a windshield, on a head updisplay (HUD), on a screen in the vehicle, on a portable network device,on a side window, or elsewhere in the vehicle. As used herein a portablenetwork device (or simply “network device”) may be a cellular phone, alaptop computer, a tablet, a wearable device, a display, etc.

The environment interactive systems are network connected, occupantconnected, interactive, instructive, personalized and entertaining andapplicable to different levels of autonomous vehicles, autonomous publictransportation (e.g. a robotaxi), shared vehicles, public vehicles, andother vehicles. Occupant connectivity refers to the obtaining ofpersonal information of occupants and performing operations based on thepersonal information and a surrounding environment and may includeproviding advertisements customized for occupant interests.Interactivity refers to the ability of an occupant to interact with theenvironment interactive systems and the surrounding environment as seenin a field of view of the occupant by providing gestures (e.g., handpointing, eye blinking, etc.). The field of view refers to an area of anenvironment external to a vehicle of an occupant and forward or to theside of the occupant, such as the field of view through a windshield orside window of the vehicle. The environment interactive systems detectthe gestures and/or contact with touchscreens. The environmentinteractive systems can be a better display platform than a smartphone.

The environment interactive systems are able to operate as tour guides,shopping assistants, and marketing tools to provide targetedadvertisements and coupons. Information and advertisements are providedbased on personal interests of the occupants. Relevant coupons based onoccupant preferences are provided and may be transferred to portablenetwork devices for immediate use. Occupant selected informationincluding object information, advertisements and coupons may be saved tothe portable network device of the occupant. Occupants can interact withthe environment interactive systems using gestures by, for example,pointing at displayed items and/or objects in a field of view of theoccupant. This may include hand movements, referred to as a handgesture, to signify tapping on the displayed items and/or objects in thefield of view. Occupants may also interact via touchscreens and/or othertraditional user input devices. By providing the stated entertainment,vehicle occupants remain active in a vehicle and spend less timeutilizing cellular phones. This can minimize the probability of anoccupant getting motion sickness in a fully autonomous vehicle.

The environment interactive systems provide a unique visual experienceto occupants including a connection to the real world via a shoppingassistant and instructive information of current surroundings of avehicle. Occupants are provided with a new shopping experience includingseamless personalized advertisements and real time marketing based on asurrounding environment, which brings occupants back to products. Costsof using autonomous vehicles and autonomous rides are able to be reducedvia ride sharing and use of the vehicles for advertising products andservices.

By being instructive, the environment interactive systems provide tovehicle occupants information relative to a current environment externalto a vehicle. The environment interactive systems provide entertainmentto vehicle occupants while commuting in a vehicle without use of a smartphone. The environment interactive systems are able to play games, suchas trivia games. The games may be played by multiple occupants within asame vehicle or within different vehicles using vehicle-to-vehicle(V-to-V) communication.

A real time mapping system is disclosed and combines an in-vehiclestereo vision system including cameras, perception control, gestureinterpretation, eye tracking, finger (or body part) tracking,geographical location tracking, object tracking and identification, etc.The real time mapping system allows an occupant of a vehicle to “click”on targets (e.g., buildings surrounding the vehicle) of interest using agesture. The occupant may directly click on real targets of interest,which are located outside the vehicle, for example, by pointing and/orwaving a finger towards the targets. Position and/or movement of one ormore eyes and/or fingers are monitored and tracked and gestureinterpretation is implemented to determine the target being selected.This is implemented while the occupant is in the vehicle, which may bestopped or moving. Information is displayed for selection by theoccupant on a HUD and/or via some other imaging device and/or system.The information may be displayed adjacent to or on detected objects,such as in front of objects (i.e. between the occupants and the objectsand in the field of view of the occupant). The phrase “adjacent to” inthis context means near and to the side of the object and in the fieldof view of the occupant. Items that are displayed on objects may bedisplayed in front of or overlaid on the objects so as to be viewed on afront surface of the objects. Items may be displayed for an object inother areas of the field of view and/or on other displays. Once aselection is made, additional information including a next level ofdetail may be displayed and this process may be continued. This realtime mapping system connects occupants to the real world surrounding thevehicle with an ergonomically advantageous interaction experience. Theexamples aid in eliminating cell phone dependence and distracteddriving.

FIG. 1 shows a vehicle entertainment access system 100 that includes avehicle 101, a distributed communications system 102, and a centraloffice 104 of a service provider. The vehicle 101 includes anenvironment interactive system (referred to also as a real time mappingsystem), a portion of which is shown and designated 106. The environmentinteractive system includes an infotainment module 108 that executes anenvironment interactive application 110. The infotainment module 108communicates with network devices 112 and a central office 104 via atelematics module 116. The network devices 112 may be implemented asdisplays, portable network devices, and/or other network devices. Thedisplays (or viewing devices) may include HUDs, laser holographicdevices, liquid crystal displays (LCDs), and/or other 2D and 3Ddisplays. A windshield and/or window may be used to display items and assuch may be referred to as a display. The telematics module 116 includesa telematics control module 118 and transceivers 120, which may includeBluetooth® transceivers, wireless local area network (WLAN)transceivers, wireless fidelity (Wi-Fi®) transceivers, cellular and/orother transceivers. The infotainment module 108 is connected to thetelematics module 116, a vehicle control module 121, and other modules124 via a bus 122 (e.g., a controller area network (CAN) bus). Theinfotainment module 108 stores occupant information 130 and preferences131 in a memory 132. Examples of occupant information and preferencesare disclosed below. As used herein the term “occupant” may be replacedwith the term “user”. As further disclosed herein, the infotainmentmodule 108 sends items to the network devices 112 for display based onthe occupant information and preferences.

The infotainment module 108 communicates with a monitoring station 140of the central office 104 via the distributed communications system 102.The monitoring station 140 may include a control module 142, atransceiver 144 and a database and/or memory 146. Occupant informationand preferences may be stored in the database and/or memory 146 andshared with the infotainment module 108. The monitoring station mayoperate as, include and/or have access to cloud-based storage. Theinfotainment module 108 may access the Internet and provide Internetconnectivity for downloading advertisements and educational informationvia the distributed communication system 102. In one embodiment, theenvironment interactive application is implemented as part of a vehiclecommunication interface module (e.g., an OnStar® module) and themonitoring station 140 provides remote services associated with thevehicle communication interface module. The vehicle communicationinterface module may be one of the other modules 124. Occupantinformation and information pertaining to objects and correspondingitems may be obtained through an internet connection or backend servicesprovided within the vehicle 101 as further described below. The backendservices may be at least partially provided by the monitoring station140 and/or other remote network device.

FIG. 2 shows the vehicle 101 incorporating a vehicle system 200including the environment interactive system 106. The vehicle 101includes the infotainment module 108 including the environmentinteractive application 110, the telematics module 116, the vehiclecontrol module 121, the memory 132, a navigation module 204, sensors206, microphones 207, a propulsion system 208, an inertial measurementmodule 210, and other vehicle systems 212.

The telematics module 116 provides wireless communication serviceswithin the vehicle 101 and wirelessly communicates with serviceproviders. The telematics module 116 may support Wi-Fi®, Bluetooth®,Bluetooth Low Energy (BLE), near-field communication (NFC), cellular,legacy (LG) transmission control protocol (TCP), long-term evolution(LTE), and/or other wireless communication and/or operate according toWi-Fi®, Bluetooth®, BLE, NFC, cellular, and/or other wirelesscommunication protocols. The telematics module 116 may include thetelematics control module 118 and a transceiver 120. The telematicscontrol module 118 may include a global positioning system (GPS) 214.The transceiver 120 includes a physical layer (PHY) module 216 and amedium access control (MAC) module 218. The PHY module 216 wirelesslycommunicates with network devices internal and external to the vehicle101. The MAC module 218 may perform pattern recognition, channeladdressing, channel access control, and filtering operations.

The infotainment module 108 may include and/or be connected to an audiosystem 220 and/or a video system 222 including one or more displays (onedisplay 224 is shown in FIG. 2 ) to provide vehicle status information,diagnostic information, prognostic information, entertainment features,and other vehicle and non-vehicle related information. The entertainmentfeatures may include: object information; business information;environment information, advertisements; coupons; game associatedinformation, data and/or questions; and/or other entertainment data. Theinfotainment module 108 may also provide entertainment features, such asallowing an occupant (or passenger) to play games within the vehicle,outputting (visibly and/or audibly) local information to the passenger,providing advertisements to the occupant, providing coupons to thepassenger and/or other entertaining features disclosed herein. Thesefeatures are provided to improve an occupant experience within thevehicle. The displays may also display (i) items captured by theexternal cameras 230 including objects detected that are external to thevehicle, and (ii) information associated with the detected objects.

The vehicle control module 121 may control operation of the vehicle 101including operation of the propulsion system 208 and other vehiclesystems 212. The memory 132 may store the occupant information 130, theoccupant preferences 131, entertainment information 223, Internet basedinformation 225, object information 226, business information 227, mapdata 228, and other data 229. The occupant (or personal) information 130may include, for example, occupant names, occupant identifiers (IDs),occupant passwords, facial recognition information, fingerprintrecognition data, eye recognition data, and other personal information.The object information 226 and the business information 227 may includenames of the objects and businesses, addresses of the businesses, agesof the objects and businesses, histories of the objects and businesses,etc. The occupant preferences 131 may include personal applications,preferred podcasts, music preferences, types of information searched,online videos watched, social media information, preferred restaurantsand/or bars, hobbies, work related information, vacation locations,items searched and/or purchased, preferred hotels, preferred airlines,preferred methods of traveling, etc. Restaurant related information mayinclude types of food preferred, coffee shops preferred, rating and costpreferences, etc. The personal information 130 and/or occupantpreferences 131 may include information related to online sources, suchas Audible®, Spotify®, Pandora®, YouTube®, etc. The personal information130 and/or occupant preferences 131 may include information related tosocial media, such as Facebook®, Instagram®, Snapshot®, etc. Thepersonal information 130 and/or occupant preferences 131 may includeinformation related to search engines and/or shopping sources, such asGoogle®, Yelp®, Amazon®, etc.

The entertainment information 223 may include some of the occupantpreferences, games and/or trivia information and/or questions related toa local environment surrounding the vehicle 101, local events, popular“things to see” in a local area, the Internet based information 225, theobject information 226, the business information 227, etc. The internetbased information 225 may include information related to websites,products, services, videos, restaurants, and/or other items searchedand/or viewed via the Internet.

The object information 226 may include fauna and/or flora information,architectural information, historical information, landmark information,building information, etc. The business information 227 may includehistorical information, information describing the business, buildinginformation, architectural information, stock information, companystatus information, etc. The map data 228 may include geographical data,location identification information, and local weather. The other data229 may include traffic information, time of day information, etc.

The navigation module 204 executes a navigation application to providenavigation services. As an example, if the vehicle 101 is an autonomousvehicle, the navigation module 204 may direct the vehicle control module121 to a certain location.

The sensors 206 may include external cameras 230, internal cameras 232,and other vehicle sensors 234 (e.g., a vehicle speed sensor). Thecameras 230, 232 may be 2-dimensional cameras, 3-dimensional cameras,depth cameras, infrared cameras, visible light cameras, and/or othersuitable cameras. The external cameras 230 and the telematics controlmodule may be used to detect objects external to the vehicle 101. Thecameras 230, 232 may include corresponding infrared (IR) sources 236. IRsources may be used on an exterior of a vehicle, when for example thereis low visibility (visibility distance is low), such as on a foggy day.In one embodiment, visible light cameras and IR cameras are used. Theexternal cameras 230 are used to monitor areas surrounding the vehicleand may be in various locations on the vehicle. The internal cameras 232are used to monitor positions of occupant eyes, monitor finger positionsof occupants, and/or determine gaze points of occupants. The positionsof the occupant eyes refer to physical locations within the vehicle 101.A gaze point refers to a direction in which eyes of an occupant arelooking and an environment location the occupant is looking at. Themicrophones 207 may be used to receive audio commands from occupants.

The propulsion system 208 may include an internal combustion engine 244,a transmission 245, and/or one or more electric motors 246 to propel thevehicle 101. The vehicle control module 121 may control the propulsionsystem 208 to cause the vehicle 101 to move to selected locations.

The inertial measurement module 210 includes inertial measurementsensors 248, such as accelerometers and/or rate sensors. The inertialmeasurement module 210, as an example, may be located at or near acenter of gravity (cg) point of a vehicle. The inertial measurementsensors 248 may be used to measure longitudinal and lateralaccelerations and/or longitudinal and lateral velocity of the vehicle101. The longitudinal and lateral accelerations and the longitudinal andlateral velocity may be integrated to determine pitch angles and rollangles of the vehicle 101 to determine orientation of the vehicle.

FIG. 3 shows a portion 300 of the vehicle system 200 of FIG. 2 and aportion 301 of the environment interactive system 106 of FIG. 2 .Although vehicle system 200 of FIG. 2 is shown as a hybrid vehiclesystem in FIGS. 2-3 , the present disclosure is also applicable tonon-hybrid vehicles, electric vehicles, fuel cell vehicles, and othertypes of vehicles including autonomous vehicles, semi-autonomousvehicles, non-autonomous vehicles, shared vehicles, non-shared vehicles,and other types of vehicles.

The vehicle system 200 includes the engine 244, which may combust anair/fuel mixture to generate drive torque. An engine control module(ECM) 306 controls the engine 244. In one embodiment, the vehicle system200 is a fully electric vehicle and the ECM 306 and the engine 244 arenot included. For example, the ECM 306 may control actuation of engineactuators, such as a throttle valve, one or more spark plugs, one ormore fuel injectors, valve actuators, camshaft phasers, an exhaust gasrecirculation (EGR) valve, one or more boost devices, and other suitableengine actuators. In some types of vehicles (e.g., electric vehicles),the engine 244 may be omitted.

The engine 244 may output torque to the transmission 245. A transmissioncontrol module (TCM) 314 controls operation of the transmission 245. Forexample, the TCM 314 may control gear selection within the transmission245 and one or more torque transfer devices (e.g., a torque converter,one or more clutches, etc.).

The vehicle system 200 may include the one or more electric motors 246shown in FIG. 2 , such as the motor 318. For example, an electric motor318 may be implemented within the transmission 245 as shown. Theelectric motor 318 may perform as either a generator or as a motor. Whenoperating as a generator, the electric motor 318 converts mechanicalenergy to electrical energy. The electrical energy may be, for example,used to charge a battery 326 via a power control device (PCD) 330. Whenoperating as a motor, the electric motor 318 generates torque that isused, for example, to supplement or replace torque output by the engine244. The vehicle may include zero or more electric motors.

A power inverter module (PIM) 334 may control the electric motor 318 andthe PCD 330. The PCD 330 applies power from the battery 326 to theelectric motor 318 based on signals from the PIM 334, and the PCD 330provides power output by the electric motor 318, for example, to thebattery 326. The PIM 334 may include, for example, an inverter.

A steering control module 340 controls steering/turning of wheels of thevehicle, for example, based on driver turning of a steering wheel withinthe vehicle and/or steering commands from one or more vehicle controlmodules. A steering wheel angle (SWA) sensor monitors rotationalposition of the steering wheel and generates a SWA signal 342 based onthe position of the steering wheel. As an example, the steering controlmodule 340 may control vehicle steering via an electronic power steering(EPS) motor 344 based on the SWA signal 342. However, the vehicle 101may include another type of steering system.

An electronic brake control module (EBCM) 350 may selectively controlbrakes 354 of the vehicle. A horn module 356 may apply power to a horn358 when a horn input 360 is in a first state. The horn 358 outputssound when power is applied to the horn 358. The horn module 356 may notapply power to the horn 358 when the horn input 360 is in a secondstate. The horn input 360 may transition from the second state to thefirst state, for example, in response to occupant application of atleast a predetermined force is applied to a horn input device (e.g.,located on the steering wheel). The horn input device may apply a biasforce such that the horn input 360 is in the second state when less thanthe predetermined force is applied to the horn input device.

Modules of the vehicle may share parameters via the bus 122, which mayinclude one or more data buses. Various parameters and data may be madeavailable by a given module to other modules via the bus 122.

Some example driver inputs include, for example, an accelerator pedalposition (APP) 366, which may be provided to the ECM 306. A brake pedalposition (BPP) 370 may be provided to the EBCM 350. A position 374 of apark, reverse, neutral, drive lever (PRNDL) may be provided to the TCM314. An ignition state 378 may be provided to a body control module(BCM) 380. For example, the ignition state 378 may be input by a drivervia an ignition key, button, or switch. At a given time, the ignitionstate 378 may be one of off, accessory, run, or crank.

The infotainment module 108 selectively provides object information,entertainment information, and other information via one or more outputdevices 384. The infotainment module 108 also selectively executes gamesand outputs trivia questions and gaming feedback via the one or moreoutput devices 384. The output devices 384 may include, for example, anyof the displays referred to herein, one or more sets of virtual reality(VR) goggles, one or more sets of augmented reality (AR) goggles, one ormore other suitable types of video output devices, one or more speakers,one or more haptic devices, and/or one or more other suitable types ofoutput devices. The infotainment module 108 outputs gaming video via theone or more displays, one or more sets of VR goggles, and/or one or moresets of AR goggles. The infotainment module 108 outputs gaming audio viathe one or more speakers. The infotainment module 108 may also outputother gaming feedback via one or more haptic devices. For example,haptic devices may be included with one or more seats, in the steeringwheel, etc.

Examples of displays may include, for example, one or more displays(e.g., on a front console) of the vehicle, one or more displays ofportable network devices, a HUD that displays information via asubstrate (e.g., windshield), one or more displays that drop downwardlyor extend upwardly to form panoramic views, and/or one or more othersuitable displays. In various implementations, one or more HUDs mayproject onto the windshield to use the windshield as a panoramicaugmented reality (AR) display.

The infotainment module 108 controls play of one or more games based onoccupant input received via one or more input devices 385, such as oneor more of the displays referred to herein, center console input devices(e.g., switches, knobs, etc.), gaming controllers, one or morejoysticks, etc. Under some circumstances, such as when the vehicle isparked, the infotainment module 108 may control gameplay based on inputfrom vehicle components (e.g., the steering wheel, brake and acceleratorpedals, horn, etc.). The infotainment module 108 may additionally oralternatively audibly and/or visibly output other information, such aspersonalized advertisements, coupons, information about points ofinterest in front of the vehicle, etc.

The infotainment module 108 may provide the stated outputs based oninputs from external and internal sensors 386, such as external andinternal cameras. For example, the infotainment module 108 may displaytext messages, text alerts, icons, objects and/or other items on adisplay via input from the external and internal sensors 386 whileoccupants are in the vehicle 101.

As another example, based on input from the external sensors, an objectdetection module 390 may detect, identify and determine informationpertaining to objects around the vehicle 101 and locations of theobjects relative to the vehicle 101. The infotainment module 108 mayinclude the object detection module 390 and adjusts items displayedbased on the determined object information.

FIG. 4 shows an interior 400 of the vehicle 101 of FIGS. 1-2 and aportion 402 of an environment forward of the vehicle 101 implementingthe environment interactive system 106 of FIG. 2 . The interior 400includes seats 404, a center console 406, a dashboard 408 includingdisplays 410, 412, and a HUD 414. The environment includes buildings416, which are visible in a field of view of occupants in the seats 404,405 through a windshield 418. The center console 406 includes a display422, interface switches and knobs 424, and a charging station 426. Aportable network device 428 may be set on the charging station 426 andbe charged. The portable network device 428 may wirelessly communicatewith the infotainment module 108 of FIGS. 1-3 via, for example, aBluetooth connection or other suitable wireless connection. The portablenetwork device 428 may include an interactive application forcommunicating with the infotainment module 108 of the vehicle 101 duringan environment interactive mode. During the environment interactivemode, occupants may interact with an environment and/or items displayedby looking at the objects in the environment and/or the displayed items,pointing at the objects and/or items, and gesturing. To gesture anobject and/or item, an occupant may point at the object and/or item,move a finger of the occupant as to “tap” on the object and/or item,touch one of the touchscreens, etc. Gesturing is further describedbelow.

The infotainment module 108 may display items on the windshield 402,over the buildings 416 and/or in other areas of the windshield, on theHUD 414, on the displays 410, 412 and/or on the display 422, asdescribed herein. The items displayed may be personalized for either twooccupants sitting respectively in the seats 404, 405. As an example, GPSand compass related information including maps, heading directions, etc.may be displayed on the display 422. The displays 410, 412 may bedeployable (i.e. movable) up to and away from the laps of occupants inthe seats 404, 405.

A sensor module 430 may be included and attached to the top of thewindshield and/or to a ceiling of the vehicle 101. The sensor module 430may include internal cameras 432. The sensor module 430 may include IRsources, such as the IR sources 236 shown in FIG. 2 . The sensor module430 may include any number of cameras and IR sources for each occupantand a corresponding one of the seats 404, 405 of that occupant. One ormore cameras and IR sources may be directed to an occupant in seat 404and another one or more cameras and IR sources may be directed to anoccupant in seat 405. The internal cameras 432 are used to monitorpositions of occupant eyes, position and movement of occupant fingersand/or hands, and/or gaze points of the occupants. The interior 400 mayinclude an audio system and wireless transceivers for transmittingsignals to and receiving signals from the portable network device 428.

The above-described external and internal cameras are positioned tocapture images and video outside of (external to) the vehicle and insideand internal to the vehicle. External forward facing cameras captureimages and video of images within a predetermined field of view (FOV) infront of the vehicle. A right side mounted camera may be used to captureimages and video within a predetermined FOV to the right of the vehicle.A left side mounted camera captures images and video within apredetermined FOV to the left of the vehicle. The external sensors mayadditionally or alternatively include various other types of sensors,such as ultrasonic sensors, radar sensors, LIDAR sensors, etc. Theexternal sensors may be used to detect distances between the vehicle andobjects, positions of objects relative the vehicle, etc.

The systems disclosed herein may be operated using numerous methods,example methods are illustrated in FIGS. 5 and 8-11 . Although thefollowing methods are shown as separate methods, one or more of themethods and/or operations from separate methods may be combined andperformed as a single method. Although the following operations of FIGS.5 and 8-11 are primarily described with respect to the implementationsof FIGS. 1-4 , the operations may be easily modified to apply to otherimplementations of the present disclosure. The operations may beiteratively performed. The operations may be performed by theenvironment interactive system 106 of FIG. 2 .

FIGS. 5A-C illustrate an entertainment interactive method. FIGS. 5A-Care collectively referred to as FIG. 5 . The method may begin at 500. At502, the infotainment module 108 via the external sensors and/orcameras, the internal sensors and/or cameras, and/or other sensorsdetects (i) a network device (e.g., a portable network device) that isnear, approaching and/or moved into the vehicle 101, and/or presence ofan occupant in the vehicle 101. The occupant may have moved into theinterior cabin of the vehicle 101.

At 504, the infotainment module 108 via the telematics module 116 mayestablish a connection with the network device if a connection has notalready been established. This may include performing a handshakeprocess.

At 505, the infotainment module 108 performs a recognition process torecognize the network device and/or occupant. This operation may betriggered by the network device being moved into the vehicle. Forexample, a computing device (e.g., cell phone) may wirelessly connectwith the vehicle, such as via an application executing on the computingdevice. The wireless connection may be, for example, using a Bluetoothtransceiver, a WiFi transceiver, or another suitable type of wirelesstransceiver. The infotainment module 108 may recognize the occupantbased on a unique identifier of the computing device. Additionally oralternatively, another type of recognition may be used. For example, theinfotainment module 108 may recognize the occupant via biometricrecognition, such as facial recognition, fingerprint recognition, speechrecognition, etc.

At 506, personal information and/or preferences pertaining to theoccupant are (i) uploaded from the network device to the infotainmentmodule 108, (ii) downloaded from the monitoring station 140 to theinfotainment module 108, and/or (iii) downloaded from anothercloud-based server, database and/or memory to the infotainment module108. Operation 506 may be performed after the network device and/or theoccupant is recognized by the infotainment module 108. This recognitionis further described below with respect to one or more of the methods ofFIGS. 8-11 .

At 508, the infotainment module 108 may perform a recognition process torecognize the occupant. This operation may be triggered by the occupantmoving into the interior of the vehicle 101. This recognition processmay be similar to the above-described recognition process and be basedon, for example, facial, fingerprint, eye, speech and/or other personalrecognition.

At 510, the infotainment module 108 accesses the personal informationand/or preferences. This may include the downloading of the personalinformation and/or preferences as described above.

At 512, sensor data is generated. This may include GPS data, compassdata, camera data, vehicle orientation data, vehicle position data,and/or other vehicle data. The external and internal cameras may captureimages of an environment around the vehicle 101 and of the occupants.The images of the occupants may be of the occupant's eyes andsurrounding areas for later determination of eye positions, fingerand/or hand positions, and/or gaze points.

At 513, the infotainment module 108 determines the location andorientation of the vehicle 101. The infotainment module 108 obtains thepresent location of the vehicle, such as from the GPS 214. Theinfotainment module 108 may also obtain a heading and/or orientation ofthe vehicle 101, such as from the GPS 214, a compass of the vehicle 101,the inertial measurement module 210, and/or images from the externalsensors and/or cameras.

At 514, the infotainment module 108 determines whether an input has beenreceived from the occupant via, for example, the corresponding networkdevice to initiate the environment interactive mode. If an input hasbeen received, operations 516 and/or 518 may be performed, otherwise themethod may end at 520 in which case no entertainment (no visible oraudible output) may be provided. In one embodiment, the environmentinteractive mode is automatically enabled when the occupant is presentin the vehicle.

At 516, the infotainment module 108 determines whether an input has beenreceived from the occupant via, for example, the corresponding networkdevice to initiate the entertainment mode. If an input has beenreceived, the entertainment mode is activated and operation 522 of FIG.5B may be performed, otherwise operation 518 may be performed. Theinfotainment module 108 may activate the entertainment modeautomatically (e.g., in response to the recognition of the passenger) orin response to receipt of occupant input (e.g., via a button, switch,etc.) to activate the entertainment mode.

At 518, the infotainment module 108 determines whether an input has beenreceived from the occupant via, for example, the corresponding networkdevice to initiate the targeted marketing mode. If an input has beenreceived, the targeted marketing mode is activated and operation 550 ofFIG. 5C may be performed, otherwise the method may end at 520. Theinfotainment module 108 may activate the entertainment modeautomatically (e.g., in response to the recognition of the passenger) orin response to receipt of occupant input (e.g., via a button, switch,etc.) to activate the entertainment mode.

At 522, an input may be received from the occupant via the correspondingnetwork device selecting the one or more types of entertainment to beenabled. The infotainment module 108 may provide one or more types ofentertainment, such as executing a game (e.g., a trivia game) at 538 oroutputting (e.g., visibly and/or audibly) local information at 524. Thetype of entertainment may be selected by the occupant and/or determinedby the infotainment module, for example, based on the occupantinformation and personal preferences. Operations 524 and/or 536 may beperformed subsequent to operation 522.

At 524, the infotainment module 108 obtains local information anddisplays environment based interactive entertaining information on oneor more displays, windshield, window, etc., as described above. Thelocal information may be obtained wirelessly from one or more remotesources (e.g., one or more websites, such as Wikipedia, one or moredatabases, etc.), the monitoring station 140, and/or from the memory 132of the vehicle. For example, the infotainment module 108 may displaylocal information of objects and/or for businesses that are presentlyvisible in front of the vehicle. The infotainment module 108 may, forexample, project information associated with a business or point ofinterest onto the windshield or into the virtual environment at thelocation of the business or point of interest. FIG. 6 includes anexample illustration of information displayed in association withbuildings, businesses (local stores, restaurants, etc.) and points ofinterest. The information may be displayed based on interests of theoccupant, which may be indicated and/or determined based on the occupantinformation and preferences.

In one embodiment, the infotainment module 108 operates as a tour guidedisplaying information and/or generating corresponding audio messages.In another embodiment, the infotainment module 108 operates as ashopping assistant and indicates stores that the occupant may beinterested in and downloads and/or provides coupons to the occupantnetwork device (e.g., cell phone or other network device).

The information displayed at 524 and during other operations disclosedherein may be based on gestures from an occupant, some examples of whichare shown in FIGS. 12-13 , voice commands, touches on a touchscreen,input via buttons, etc.

As another example, the personal information may include a shoppinglist, which may be uploaded to the memory 132 via one of the networkdevices. The infotainment module 108 may display items identifyingrecommended stores where the products on the shopping list are availableand as stated below corresponding coupons for discounts on the products.For example, the infotainment module 108 may display a message stating“the shoes you are looking for are available at this store, buy todayfor additional 25% discount” or “There is a 2-for-1 sale at items atthis grocery store.”

In addition to a visual display, the infotainment module 108 may audiblyoutput local information regarding buildings, businesses and/or pointsof interest via one or more speakers. Speakers may be used, for example,for complementary information to avoid saturating a field of view. FIG.6 is an example view illustrating displaying interactive items in frontof objects in a field of view of an occupant. The items may be icons,text and/or other selectable items. The occupant may select the items toobtain additional information which may be displayed in front of theobject and/or near the object in the field of view of the occupant.

At 526 of FIG. 5B, the infotainment module 108 may determine a type ofeach object external to the vehicle 101 and a location of each objectrelative to the vehicle 101. Examples of types of objects include, forexample, vehicles, pedestrians, cyclists, animals, buildings, etc. Theinfotainment module 108 may determine the types of the objects using anobject detection algorithm. The infotainment module 108 may determinethe locations of the objects, for example, based on the FOV of thecameras, the location of the object in the pixels of an image, anddimensions (e.g., pixelwise) of the object in the image. Trafficstructures (e.g., buildings, roads, mountains, etc.) are identified.

At 528, the infotainment module 108 may display available channelsassociated with detected objects including structures, buildings,businesses, etc. For example, if a business in the field of view of theoccupant is a television studio, then the one or more channelsassociated with that business may be displayed.

At 530, the infotainment module 108 may determine whether an input hasbeen received from the occupant indicating selection of one of thedisplayed channels. If an input has been received, operation 532 may beperformed and the channel may be displayed for viewing by the occupant.As an example, the occupant may select a history channel to watch. If aninput has not been received, then operation 534 may be performed.

At 534, the infotainment module 108 may determine whether an indicationhas been received to stop displaying the entertaining information and/orcontent. This indication may be in the form of an input from theoccupant, or from another controller and/or sensor of the vehicleindicating that the vehicle has stopped, the occupant has left thevehicle, an alert interruption has occurred and/or some other type ofindication. If an indication has been received, the method may end at548, otherwise operation 524 may be performed.

At 536, the infotainment module 108 may determine whether an input hasbeen received to play a game, such as a trivia game. An item may bedisplayed for the occupant to select to start the game. The item may bean icon, a list of games to select from, a message, a question asking ifthe occupant would like to play a game and/or a particular game, etc. Asan example, the trivia game may include the infotainment module 108displaying questions related to one or more objects in the field of viewof the occupant. Questions provided to one occupant may be differentthan questions asked of a different occupant. Questions provided to eachoccupant may be based on the objects in the field of view of thatoccupant, the occupant information, and/or the preferences of theoccupant. If an input is received to play a game, operation 538 isperformed, otherwise operation 524 may be performed. At 538, theinfotainment module 108 may play the environment interactiveentertainment game selected at 536.

At 540, the infotainment module 108 may determine whether a manual orautomatic mode has been selected. In one embodiment, this selection ismade by the occupant. If the manual mode is selected, operation 542 maybe performed, which includes displaying information and/or triviaquestions based on gestures and/or eye movement, which is tracked by oneor more of the internal cameras 232. Gestures may include the occupantmoving one or more hands and/or fingers, to select, tap on, point at,and/or otherwise indicate an answer, a response, and/or provide aninput. Other types of gestures may be used. Voice responses may also beused. If the automatic mode has been selected, operation 544 may beperformed, which includes displaying information and/or trivia questionsbased on occupant information and/or personal preferences.

At 546, the infotainment module 108 may determine whether an indicationhas been received to stop playing the selected game. If an indicationhas been received, the method may end at 548, otherwise operation 538may be performed.

At 550 of FIG. 5C, the infotainment module 108 may display personalizedadvertisements to the occupant. The infotainment module 108 maydetermine which advertisements to display—based on the occupantinformation. Different advertisements may be displayed for differentoccupants in the vehicle based on the information and preferences of theoccupants. The infotainment module 108 may determine whichadvertisements to display further based on the present location of thevehicle. For example, the infotainment module 108 may displayadvertisements for businesses that are presently visible in front of thevehicle. Additionally or alternatively, the infotainment module 108 maydisplay advertisements similar to and/or different than randombillboards that the vehicle is passing. FIG. 7 includes an exampleillustration of an advertisement displayed in place of a billboard. Theoccupant may select the advertisement to obtain additional informationregarding the advertised product and/or service. In addition to a visualdisplay, the infotainment module 108 may audibly output informationregarding advertisements via one or more speakers.

At 552 of FIG. 5C, the infotainment module 108 may display targetedcoupons targeting each of the occupants and the interests andpreferences of the occupants. The coupons may be displayed in relationto the displayed advertisements and/or when the occupant taps on,gestures, points at and/or otherwise selects the advertisementdisplayed. An example coupon in shown in FIG. 7 in association with anadvertisement. The advertisement may be an advertisement on a billboardor an advertisement provided in place of a billboard. The occupant mayselect the coupon to have the coupon downloaded and/or to receiveadditional information related to the coupon. The advertisements andcoupons may be obtained, for example, from one or more remote sources,examples of which are disclosed above, and/or from the memory 132. FIG.7 shows a head up display illustrating inclusion of an advertisement anda corresponding coupon (or discount).

At 554 of FIG. 5C, the infotainment module 108 may transmit and/ordownload advertisements, coupons and/or other business relatedinformation to the network devices of the occupants.

At 556 of FIG. 5C, the infotainment module 108 may monitor if thevehicle stops at a place recommended by one of the advertisements. Whenthe vehicle makes a stop at a location (e.g., a business or a point ofinterest) for at least a predetermined period (e.g., more than 1minute), the infotainment module 108 may transmit advertisements and/orcoupons to the network devices of the occupants.

At 558 of FIG. 5C, the infotainment module 108 may determine whether toperform a survey. If yes, operation 560 is performed, otherwiseoperation 566 may be performed.

At 560 of FIG. 5C, the infotainment module 108 may transmitadvertisements and/or coupons to the network devices of the occupantsand display one or more occupant interfaces requesting occupant inputassociated with a survey. Questions may be displayed to enable theoccupant to rate the business (e.g., restaurant, store, etc.) at thelocation or point of interest. The questions may be displayed on one ormore displays, windshield, and/or window as described above. Aninteractive, simple and quick rating system message may be displayed. Anexample quick rating message 561 is shown in FIG. 4 . In FIG. 4 a hand563 of an occupant is shown pointing at one of the answers, which isbeing selected. The infotainment module 108 via the internal cameras 232detects this selection and stores the answer. The location may be aplanned stop of a navigation route or identified based on the vehiclebeing stopped at the location for at least the predetermined period. At562, a survey and/or rating questions may be displayed. The occupantsmay provide answers and/or rating responses via the occupantinterface(s). The infotainment module 108 may remove the occupantinterface at a predetermined period after the vehicle begins moving awayfrom the location. The infotainment module 108 may determine thebusiness or point of interest at the location based on the navigationroute or in another suitable manner.

At 564 of FIG. 5C, the infotainment module 108 may store the answersand/or responses in the memory 132 and/or upload the answers and/orresponses to one or more of the remote sources.

At 566, the infotainment module 108 may determine an indication has beenreceived to stop targeted marketing. If an indication has been received,the method may end at 568, otherwise operation 550 may be performed.

The above-described method includes the infotainment module 108determining whether to provide entertainment interactively (in responseto occupant input), automatically (with or without occupant input), orto execute a game. For example, in response to occupant input, theinfotainment module 108 may provide entertainment informationinteractively. The infotainment module 108 may, for example, output(audibly or visibly) local information to the occupant in response toreceipt of occupant input regarding a location (e.g., a business or apoint of interest). In an embodiment, the infotainment module 108 mayprovide entertainment information automatically. The infotainment module108 may, for example, output (audibly or visibly) local information tothe occupant based on locations (e.g., businesses or points of interest)located in front of the vehicle. The infotainment module 108 may audiblyoutput information via one or more speakers and display information, forexample, via one or more HUDs and/or one or more other displays.

During the above-described method, the infotainment module 108 mayadjust and/or modulate the amount of information being displayed and therate at which the information is being displayed based on vehicle speed.If the vehicle 101 is stopped, due to for example a traffic light,congested traffic, vehicle charging, etc., more information may bedisplayed than when traveling. The content of the information may dependon the time of day (e.g., restaurant information may be displayed duringmeal times). The described features may be implemented for passengers,not the driver of a conventional (non-autonomous) vehicle. In oneembodiment, limited information is provided for the driver of thenon-autonomous vehicle.

FIG. 8 shows an environment interactive method including eye positionand finger position tracking and displaying of object relatedinformation. The method may begin at 800. At 802, the infotainmentmodule 108 receives sensor data including data indicating eye trackinginformation and finger tracking information of an occupant. This mayinclude camera data indicating positions of one or more eyes and one ormore fingers of the occupant within an interior of the vehicle

At 804, the infotainment module 108 determines the positions of the oneor more eyes and the one or more fingers based on the data received at802 and relative to a reference point within the interior of thevehicle. This allows the position information to be compared toenvironment data and aids in determining what the occupant is looking atand/or is pointing at. At 806, the infotainment module 108 receives GPSand/or map data indicating a location of the vehicle.

At 808, the infotainment module 108 coordinates the positions of the oneor more eyes and/or the positions of the one or more fingers with thevehicle location information and relative to the surroundingenvironment. At 810, the infotainment module 108 determines the objector item being selected by the occupant based on results of thecoordination performed at 808. This may be based on whether the occupantis looking at a certain location for longer than a predetermined periodof time. In one embodiment, when the occupant is looking at an objectfor more than a predetermined period of time, then it is deemed that theoccupant is intending to select the object.

At 812, the infotainment module 108 displays information for a selectedobject or item which may include displaying the information on adisplay, windshield and/or other windows. Examples of this are shown inFIGS. 6, 7, 12-15 .

At 814, the infotainment module 108 determines whether to continue inthe environment interactive mode. If yes, operations 802 and 806 may beperformed, otherwise the method may end at 816.

FIG. 9 shows a personalized environment interactive method includingmulti-camera eye position and finger position tracking. The method maybegin at 900. At 902, the infotainment module 108 receives an indicationto start the environment interactive mode. This may include the occupantproviding an on/off input such as a gesture, a voice command, a touch ona touchscreen, a push of a button, etc. As another example, a gesturemay be a blinking eye.

At 904, the infotainment module 108 performs a login procedure. This mayinclude performing facial recognition, fingerprint recognition, eyerecognition, and/or other personal recognition procedure to recognizethe occupant. The occupant may enter and/or provide personal informationand/or personal information may be automatically downloaded from aportable network device (e.g., cell phone) of the occupant when theoccupant enters the vehicle. If the occupant is not recognized, theoccupant may be logged in as a new user and personal information and/orpreferences may be downloaded. The login procedure is performed torecognize the occupant. No credential information is needed for thisprocess, but such information may be provided. In one embodiment, theinformation collected is sent to the central monitoring station 140 ofFIG. 1 and the central monitoring station responds with informationassociated with recognizing the occupant. The central monitoring stationmay search the database and/or memory 146 for information pertaining tothe occupant to identify the occupant and/or personal information and/orpreferences of the occupant.

At 906, the infotainment module 108 determines occupant preferencesbased on the information collected and/or stored in association with theoccupant. During the login procedure, preferences (or choices) of theoccupant may be determined including, for example, tourist information,shopping mall information, grocery store information, restaurantinformation, etc. including corresponding likes and dislikes.

At 908, the infotainment module 108 receives captured images of theenvironment surrounding the vehicle and images of the inside of thevehicle. This may be done using the cameras 230, 232 of FIG. 2 . In oneembodiment, two or more of the internal cameras 232 are used to monitorone or more eyes and one or more fingers of the occupant. At 910, theinfotainment module 108 recognizes and tracks the eyes and fingers.

At 912, the infotainment module 108 determines locations of the eyes andfingers of the occupant. The locations may be determined relative to areference point in the vehicle. The reference point may be locatedanywhere in the vehicle and may refer to a fixed point in the vehicle.The infotainment module 108 may perform triangulation using data fromtwo or more cameras to determine the locations of the eyes and fingers.At 914, the infotainment module 108 determines the geographical locationof the vehicle. The location of the vehicle may be determined based onthe GPS data, IMU data and/or map data. This may include determining theheading and/or orientation of the vehicle.

At 915, the infotainment module 108 may determine the objects in thefield of view of the occupant. The field of view may be determined basedon the locations of the eyes of the occupant, the location of thevehicle and/or the exterior captured images. This may includeidentifying and determining the locations of the objects.

At 916, the infotainment module 108 coordinates the locations of theeyes and fingers of the occupant with the geographical location of thevehicle and the objects in the field of view of the occupant. This mayinclude determining the looking direction of the occupant and/or thepointing direction of the finger of the occupant to determine whichobjects the occupant is looking and/or pointing at.

At 918, the infotainment module 108 may determine one or more of theobjects being selected by the occupant based on the locations of theeyes and the fingers of the occupant. An example of an occupant pointingat a particular building is shown in FIG. 12 . The looking direction andthe pointing direction of the occupant are mapped to the exteriorenvironment and corresponding objects for target determination. As anexample, this may include determining which building the occupant islooking at and/or pointing at in a series of multiple buildings. Theinfotainment module 108 may interpret a gesture provided by the occupantwith the occupant's hand and/or other body part. The coordination may beimplemented by, for example, the monitoring station 140 of FIG. 1 .Collected data may be forwarded to the monitoring station 140 and themonitoring station may provide feedback to the infotainment module 108.

At 919, the infotainment module 108 may request confirmation of theselection made. This may include displaying a confirmation window, icon,or other confirmation item in the field of view of the occupant and theoccupant selecting the item. At 920, the infotainment module 108 maydetermine whether a confirmation is received. If a confirmation isreceived, operation 921 may be performed, otherwise operation 922 may beperformed.

At 921, the infotainment module 108 may display information for theselected one or more objects on a HUD, screen, windshield, sidewindow(s) and/or other display. An example of information beingdisplayed is shown in FIG. 13 . FIG. 13 shows an interior of a vehicleillustrating eye and finger tracking and occupant selection of an objectin a field of view.

At 922 of FIG. 9 , the infotainment module 108 may determine whether theentertainment mode is activated. If yes, operation 522 of FIG. 5B may beperformed, otherwise operation 924 may be performed.

At 924, the infotainment module 108 may determine whether to continueoperating in the environment interactive mode. If yes, operation 908 maybe performed, otherwise the method may end at 926.

FIG. 10 shows another personalized environment interactive methodincluding multi-camera eye position and finger position tracking fordisplay of targeted marketing information. The method may begin at 1000.At 1002, the infotainment module 108 receives an indication to start theenvironment interactive mode. This may include the occupant providing anon/off input such as a gesture, a voice command, a touch on atouchscreen, a push of a button, etc.

At 1003, the infotainment module 108 may determine the geographicallocation of the vehicle. The location of the vehicle may be determinedbased on the GPS data, IMU data and/or map data. This may includedetermining the heading and/or orientation of the vehicle.

At 1004, the infotainment module 108 performs a login procedure similarto the above described operation 904. This procedure may be performedbased on the location of the vehicle. For example, the types of personalinformation provided may be based on the location of the vehicle. At1006, the infotainment module 108 determines occupant preferences basedon the information collected and/or stored in association with theoccupant as described above for operation 906.

At 1008, the infotainment module 108 may determine the geographicallocation of the vehicle if not already determined for the currentiteration of operations 1008, 1010, 1012, 1014, 1016, 1018-1020, 1022,1024, 1026, 1028, 1030, and 1032. The location of the vehicle may bedetermined based on the GPS data, IMU data and/or map data. This mayinclude determining the heading and/or orientation of the vehicle.

At 1010, the infotainment module 108 may determine targeted informationbased on the geographical location of the vehicle, the occupantinformation and/or the occupant preferences. This information may beshared with and/or provided by the monitoring station 140 of FIG. 1 .The targeted information may refer to information that would be ofinterest to the occupant.

At 1012, the infotainment module 108 may display the targetedinformation corresponding to the objects within the field of view of theoccupant and/or nearby on a HUD, screen, windshield, side window(s)and/or other display. The infotainment module 108 may automaticallydisplay targeted information and/or provide certain information based onoccupant information and preferences. As another example, the occupantinformation and/or preferences may indicate that the occupant likeshistory, museums, beaches and/or parks and provide information regardingobjects and/or areas in the nearby exterior environment related to thesetopics, places, and/or items.

At 1014, the infotainment module 108 receives captured images of theenvironment surrounding the vehicle and images of the inside of thevehicle. This may be done using the cameras 230, 232 of FIG. 2 . In oneembodiment, two or more of the internal cameras 232 are used to monitorone or more eyes and one or more fingers of the occupant. At 1016, theinfotainment module 108 recognizes and tracks the eyes and fingers ofthe occupant. The infotainment module 108 performs tasks involvingperception to recognize eyes and fingers based on images from two ormore cameras. When outputs from two interior cameras are used, this isreferred to as stereo imaging. Two example interior cameras are shown inFIG. 12 .

At 1018, the infotainment module 108 determines locations of the eyesand fingers of the occupant. This may be relative to a reference pointin the vehicle. The reference point may be located anywhere in thevehicle and may refer to a fixed point in the vehicle. Triangulation maybe performed using two or more cameras to determine the locations of theeyes and fingers.

At 1019, the infotainment module 108 may determine the objects in thefield of view of the occupant based on the locations of the eyes of theoccupant, the location of the vehicle and/or the exterior capturedimages. This may include identifying and determining the locations ofthe objects.

At 1020, the infotainment module 108 coordinates the locations of theeyes and fingers of the occupant with the geographical location of thevehicle and the objects in the field of view of the occupant. This mayinclude determining the looking direction of the occupant and/or thepointing direction of the finger of the occupant to determine whichobjects the occupant is looking and/or pointing at.

At 1022, the infotainment module 108 may determine one or more of theobjects being selected by the occupant based on the locations of theeyes and the fingers of the occupant. The looking direction and thepointing direction of the occupant are mapped to the exteriorenvironment and corresponding objects for target determination. Thisdetermination may be based on known locations of a display, screen,windshield, window(s), etc. where items are displayed. This operationmay be similar to operation 918. FIG. 15 shows an example of targetedinformation being displayed and a certain portion of the informationbeing selected by the occupant looking and pointing at the information,which is displayed in front of the object in the field of view of theoccupant.

At 1024, the infotainment module 108 may request confirmation of theselection made. This may include displaying a confirmation window, icon,or other confirmation item in the field of view of the occupant and theoccupant selecting the item. At 1026, the infotainment module 108 maydetermine whether a confirmation is received. If a confirmation isreceived, operation 1028 may be performed, otherwise operation 1030 maybe performed.

At 1028, the infotainment module 108 may display on a HUD, screen,windshield, side window(s) and/or other display a confirmation messageand/or send a confirmation message to a network device associated withthe occupant. In one embodiment, an email message is sent indicating aconfirmed deal offered by a business and accepted by the occupant.Navigation instructions may be provided to direct the occupant and/orthe vehicle to a certain location, business, etc. based on, for example,the targeted information and/or other related information selected bythe occupant. The navigation instructions may be displayed on the HUD,screen, windshield, side window(s) and/or other display. In anotherembodiment, a confirmation message of a destination is shown and theoccupant confirms the destination with a gesture, as shown in FIG. 14 .In this example, the gesture is a “thumbs up” gesture. The occupant mayalso reject or defer sending the vehicle to a particular destinationwith one or more other gestures and/or inputs.

At 1030, the infotainment module 108 may determine whether an offer hasbeen received. If yes, operation 1008 is performed, otherwise operation1032 is performed. At 1032, the infotainment module 108 may determinewhether to continue operating in the environment interactive mode. Ifyes, operation 1008 is performed, otherwise the method may end at 1034.

FIG. 11 shows another personalized environment interactive methodincluding one or more cameras for monitoring the occupant gaze point andfinger position for display of targeted marketing information. Themethod may begin at 1100. At 1102, the infotainment module 108 receivesan indication to start the environment interactive mode. This mayinclude the occupant providing an on/off input such as a gesture, avoice command, a touch on a touchscreen, a push of a button, etc.

At 1103, the infotainment module 108 may determine the geographicallocation of the vehicle. The location of the vehicle may be determinedbased on the GPS data, IMU data and/or map data. This may includedetermining the heading and/or orientation of the vehicle.

At 1104, the infotainment module 108 performs a login procedure similarto operation 904 above described. This procedure may be performed basedon the location of the vehicle. For example, the types of personalinformation provided may be based on the location of the vehicle. At1106, the infotainment module 108 determines occupant preferences basedon the information collected and/or stored in association with theoccupant as described above for operation 906.

At 1108, the infotainment module 108 may determine the geographicallocation of the vehicle if not already determined for the currentiteration of operations 1108, 1110, 1112, 1114, 1116, 1118, 1120-1122,1124, 1126, 1128, 1130, 1132 and 1134. The location of the vehicle maybe determined based on the GPS data, IMU data and/or map data.

At 1110, the infotainment module 108 may determine targeted informationbased on the geographical location of the vehicle, the occupantinformation and/or the occupant preferences. This information may beshared with and/or provided by the monitoring station 140 of FIG. 1 .The targeted information may refer to information that would be ofinterest to the occupant.

At 1112, the infotainment module 108 may display the targetedinformation corresponding to the objects within the field of view of theoccupant and/or nearby on a HUD, screen, windshield, side window(s)and/or other display. The infotainment module 108 may automaticallydisplay targeted information and/or provide certain information based onoccupant information and preferences. As another example, the occupantinformation and/or preferences may indicate that the occupant likeshistory, museums, beaches and/or parks and provide information regardingobjects and/or areas in the nearby exterior environment related to thesetopics, places, and/or items.

At 1114, the infotainment module 108 creates a pattern of near-infraredlight on one or more eyes of the occupant. One of the IR light sources236 shines IR light at one or more eyes of the occupant and one of theinternal cameras 232 captures images of the one or more eyes at 1116based on IR light reflected off of the one or more eyes. In oneembodiment, only a single interior camera is used to capture theseimages of the occupant. At 1116, the infotainment module 108 receivescaptured images of the environment surrounding the vehicle and images ofthe inside of the vehicle. This may be done using the cameras 230, 232of FIG. 2 .

At 1118, the infotainment module 108 determines a position and a gazepoint of each eye being monitored. In one embodiment, the irises of theeyes are tracked using an iris tracking algorithm.

At 1120, the infotainment module 108 determines positions of the fingersof the occupant based on images from the same single camera used totrack the eyes or based on images from another camera. In oneembodiment, a first camera is designated for tracking eyes and anothercamera is dedicated for tracking fingers.

At 1121, the infotainment module 108 may determine the objects in thefield of view of the occupant based on the locations of the eyes and/orthe looking direction of the irises of the eyes of the occupant, thelocation of the vehicle and/or the exterior captured images. This mayinclude identifying and determining the locations of the objects.

At 1122, the infotainment module 108 coordinates the locations of theeyes and fingers of the occupant with the geographical location of thevehicle and the objects in the field of view of the occupant. This mayinclude determining the looking direction of the occupant and/or thepointing direction of the finger of the occupant. To determine whichobjects the occupant is looking and/or pointing at.

At 1124, the infotainment module 108 may determine one or more of theobjects being selected by the occupant based on the locations of theeyes, the looking direction of the irises, and the fingers of theoccupant. The looking direction and the pointing direction of theoccupant are mapped to the exterior environment and correspondingobjects for target determination. This determination may be based onknown locations of a display, screen, windshield, window, etc. whereitems are displayed. This operation may be similar to operation 918.

At 1126, the infotainment module 108 may request confirmation of theselection made. This may include displaying a confirmation window, icon,or other confirmation item in the field of view of the occupant and theoccupant selecting the item. At 1128, the infotainment module 108 maydetermine whether a confirmation is received. The infotainment module108 may interpret the gesture (e.g., hand gesture) or voice responseprovided by the occupant to confirm whether the IR based eye tracking onthe gaze point of the occupant for selecting (i) a target outside thevehicle, or (ii) information or item displayed in the field of viewand/or on a HUD, screen, windshield, side window(s) and/or other displayis accurate. If a confirmation is received, operation 1130 may beperformed, otherwise operation 1132 may be performed.

At 1130, the infotainment module 108 may display on a HUD, screen,windshield, side window(s) and/or other display a confirmation messageand/or send a confirmation message to a network device associated withthe occupant. In one embodiment, an email message is sent indicating aconfirmed deal offered by a business and accepted by the occupant.Navigation instructions may be provided to direct the occupant and/orthe vehicle to a certain location, business, etc. based on, for example,the targeted information and/or other related information selected bythe occupant. The navigation instructions may be displayed on the HUD,screen, windshield, side window(s) and/or other display. In anotherembodiment, a confirmation message of a destination is shown and theoccupant confirms the destination with a gesture, as shown in FIG. 14 .

At 1132, the infotainment module 108 may determine whether an offer hasbeen received. If yes, operation 1108 is performed, otherwise operation1134 is performed. At 1134, the infotainment module 108 may determinewhether to continue operating in the environment interactive mode. Ifyes, operation 1108 is performed, otherwise the method may end at 1136.

The above-described operations are meant to be illustrative examples.The operations may be performed sequentially, synchronously,simultaneously, continuously, during overlapping time periods or in adifferent order depending upon the application. Also, any of theoperations may not be performed or skipped depending on theimplementation and/or sequence of events.

The foregoing description is merely illustrative in nature and is in noway intended to limit the disclosure, its application, or uses. Thebroad teachings of the disclosure can be implemented in a variety offorms. Therefore, while this disclosure includes particular examples,the true scope of the disclosure should not be so limited since othermodifications will become apparent upon a study of the drawings, thespecification, and the following claims. It should be understood thatone or more steps within a method may be executed in a different order(or concurrently) without altering the principles of the presentdisclosure. Further, although each of the embodiments is described aboveas having certain features, any one or more of those features describedwith respect to any embodiment of the disclosure can be implemented inand/or combined with features of any of the other embodiments, even ifthat combination is not explicitly described. In other words, thedescribed embodiments are not mutually exclusive, and permutations ofone or more embodiments with one another remain within the scope of thisdisclosure.

Spatial and functional relationships between elements (for example,between modules, circuit elements, semiconductor layers, etc.) aredescribed using various terms, including “connected,” “engaged,”“coupled,” “adjacent,” “next to,” “on top of,” “above,” “below,” and“disposed.” Unless explicitly described as being “direct,” when arelationship between first and second elements is described in the abovedisclosure, that relationship can be a direct relationship where noother intervening elements are present between the first and secondelements, but can also be an indirect relationship where one or moreintervening elements are present (either spatially or functionally)between the first and second elements. As used herein, the phrase atleast one of A, B, and C should be construed to mean a logical (A OR BOR C), using a non-exclusive logical OR, and should not be construed tomean “at least one of A, at least one of B, and at least one of C.”

In the figures, the direction of an arrow, as indicated by thearrowhead, generally demonstrates the flow of information (such as dataor instructions) that is of interest to the illustration. For example,when element A and element B exchange a variety of information butinformation transmitted from element A to element B is relevant to theillustration, the arrow may point from element A to element B. Thisunidirectional arrow does not imply that no other information istransmitted from element B to element A. Further, for information sentfrom element A to element B, element B may send requests for, or receiptacknowledgements of, the information to element A.

In this application, including the definitions below, the term “module”or the term “controller” may be replaced with the term “circuit.” Theterm “module” may refer to, be part of, or include: an ApplicationSpecific Integrated Circuit (ASIC); a digital, analog, or mixedanalog/digital discrete circuit; a digital, analog, or mixedanalog/digital integrated circuit; a combinational logic circuit; afield programmable gate array (FPGA); a processor circuit (shared,dedicated, or group) that executes code; a memory circuit (shared,dedicated, or group) that stores code executed by the processor circuit;other suitable hardware components that provide the describedfunctionality; or a combination of some or all of the above, such as ina system-on-chip.

The module may include one or more interface circuits. In some examples,the interface circuits may include wired or wireless interfaces that areconnected to a local area network (LAN), the Internet, a wide areanetwork (WAN), or combinations thereof. The functionality of any givenmodule of the present disclosure may be distributed among multiplemodules that are connected via interface circuits. For example, multiplemodules may allow load balancing. In a further example, a server (alsoknown as remote, or cloud) module may accomplish some functionality onbehalf of a client module.

The term code, as used above, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes, datastructures, and/or objects. The term shared processor circuitencompasses a single processor circuit that executes some or all codefrom multiple modules. The term group processor circuit encompasses aprocessor circuit that, in combination with additional processorcircuits, executes some or all code from one or more modules. Referencesto multiple processor circuits encompass multiple processor circuits ondiscrete dies, multiple processor circuits on a single die, multiplecores of a single processor circuit, multiple threads of a singleprocessor circuit, or a combination of the above. The term shared memorycircuit encompasses a single memory circuit that stores some or all codefrom multiple modules. The term group memory circuit encompasses amemory circuit that, in combination with additional memories, storessome or all code from one or more modules.

The term memory circuit is a subset of the term computer-readablemedium. The term computer-readable medium, as used herein, does notencompass transitory electrical or electromagnetic signals propagatingthrough a medium (such as on a carrier wave); the term computer-readablemedium may therefore be considered tangible and non-transitory.Non-limiting examples of a non-transitory, tangible computer-readablemedium are nonvolatile memory circuits (such as a flash memory circuit,an erasable programmable read-only memory circuit, or a mask read-onlymemory circuit), volatile memory circuits (such as a static randomaccess memory circuit or a dynamic random access memory circuit),magnetic storage media (such as an analog or digital magnetic tape or ahard disk drive), and optical storage media (such as a CD, a DVD, or aBlu-ray Disc).

The apparatuses and methods described in this application may bepartially or fully implemented by a special purpose computer created byconfiguring a general purpose computer to execute one or more particularfunctions embodied in computer programs. The functional blocks,flowchart components, and other elements described above serve assoftware specifications, which can be translated into the computerprograms by the routine work of a skilled technician or programmer.

The computer programs include processor-executable instructions that arestored on at least one non-transitory, tangible computer-readablemedium. The computer programs may also include or rely on stored data.The computer programs may encompass a basic input/output system (BIOS)that interacts with hardware of the special purpose computer, devicedrivers that interact with particular devices of the special purposecomputer, one or more operating systems, occupant applications,background services, background applications, etc.

The computer programs may include: (i) descriptive text to be parsed,such as HTML (hypertext markup language), XML (extensible markuplanguage), or JSON (JavaScript Object Notation) (ii) assembly code,(iii) object code generated from source code by a compiler, (iv) sourcecode for execution by an interpreter, (v) source code for compilationand execution by a just-in-time compiler, etc. As examples only, sourcecode may be written using syntax from languages including C, C++, C#,Objective-C, Swift, Haskell, Go, SQL, R, Lisp, Java®, Fortran, Perl,Pascal, Curl, OCaml, Javascript®, HTML5 (Hypertext Markup Language 5threvision), Ada, ASP (Active Server Pages), PHP (PHP: HypertextPreprocessor), Scala, Eiffel, Smalltalk, Erlang, Ruby, Flash®, VisualBasic®, Lua, MATLAB, SIMULINK, and Python®.

What is claimed is:
 1. An environment interactive system for a vehicle,the environment interactive system comprising: a first camera configuredto capture first images of an area external to the vehicle; a secondcamera configured to capture second images of an area interior to thevehicle; a telematics control module configured to determine ageographical location of the vehicle; a viewing device configured todisplay items (i) at least one of adjacent to, on or in front ofdetected objects, which are external to the vehicle, and (ii) in a fieldof view of an occupant, wherein the items include informationcorresponding to the detected objects; and an infotainment moduleconfigured to detect the objects based on the first images and thegeographical location of the vehicle, based on the second images, trackat least one of a location of an eye of the occupant or a location of abody part of the occupant other than the eye, based on the location ofthe eye and the location of the body part, display the items via theviewing device in the field of view of the occupant, receive informationand preferences of the occupant, display an item for the occupant toselect on the viewing device corresponding to starting a game, and basedon the location of the eye and the location of the body part, start thegame.
 2. The environment interactive system of claim 1, furthercomprising a third camera configured to capture third images of aninterior of the vehicle, wherein the infotainment module is configuredto, based on the second images and the third images, track the at leastone of the location of the eye of the occupant or the body part of theoccupant.
 3. The environment interactive system of claim 1, wherein: theinfotainment module is configured to determine a gaze point of the eyeof the occupant; and based on the location of the gaze point and thelocation of the body part, display the items via the viewing device inthe field of view of the occupant.
 4. The environment interactive systemof claim 1, the infotainment module is configured to, based on at leastone of the location of the eye or the location of the body part, displaythe items in front of the objects and in the field of view of theoccupant.
 5. The environment interactive system of claim 1, wherein theinfotainment module is configured to: receive information andpreferences of the occupant; and based on the information andpreferences, display information about certain ones of the objects inthe field of view of the occupant.
 6. The environment interactive systemof claim 1, wherein the infotainment module is configured to: receiveinformation and preferences of the occupant; and based on theinformation and preferences, display at least one of website pages orapplications related to certain ones of the objects in the field of viewof the occupant.
 7. The environment interactive system of claim 1,wherein the infotainment module is configured to: receive informationand preferences of the occupant; and based on the information andpreferences, display at least one of targeted advertisements or couponsrelated to certain ones of the objects in the field of view of theoccupant.
 8. The environment interactive system of claim 1, wherein thegame includes displaying questions in the field of view of the occupantfor the occupant to read and answer.
 9. The environment interactivesystem of claim 1, wherein the infotainment module is configured to:determine if the vehicle has stopped at a recommended location of abusiness; and display in the field of view of the occupant a surveyquestion for the occupant to answer pertaining to the business.
 10. Theenvironment interactive system of claim 1, wherein the infotainmentmodule is configured to: detect when the occupant selects one of theobjects by gesturing at the selected one of the objects; and displayinformation pertaining to the selected one of the objects.
 11. Theenvironment interactive system of claim 1, wherein the infotainmentmodule is configured to: display information on a head up displaypertaining to one of the objects; and based on the at least one of thelocation of the eye or the location of the body part, detect when theoccupant selects the one of the objects.
 12. The environment interactivesystem of claim 1, wherein the infotainment module is configured to:interpret a gesture made by the occupant with regards to at least one ofone of the objects or one of the items; and perform an action inresponse to the gesture including enabling selection of at least one ofthe items or the objects.
 13. The environment interactive system ofclaim 1, wherein the infotainment module is configured to interpret agesture or voice response provided by the occupant to confirm whethereye tracking of a gaze point of the occupant is accurate.
 14. Theenvironment interactive system of claim 1, wherein the game is anenvironment interactive game played in an augmented reality.
 15. Theenvironment interactive system of claim 1, wherein the game includesdisplaying trivia questions, via the viewing device, related to at leastone of an environment external to the vehicle and an object in theenvironment external to the vehicle.
 16. The environment interactivesystem of claim 15, wherein the game includes feedback regarding answersto the trivia questions via at least one of an audible device and ahaptic device.
 17. An environment interactive method for an occupant ofa vehicle, the environment interactive method comprising: capturingfirst images of an area external to the vehicle; capturing second imagesof an area interior to the vehicle; determining a geographical locationof the vehicle; displaying items (i) between detected objects and theoccupant, and (ii) in a field of view of the occupant, wherein the itemscorrespond to the detected objects; detecting the objects based on thefirst images and the geographical location of the vehicle; based on thesecond images, tracking a location of an eye of the occupant and alocation of a body part of the occupant other than the eye; based on thelocation of the eye and the location of the body part, displaying theitems via a viewing device in the field of view of the occupant, whereinthe viewing device is configured, based on user preferences, to displaytargeted information for one or more businesses at one or morelocations; detecting a first one or more gestures by the occupant forselection of one of the one or more locations; based on a second one ormore gestures from the occupant, display via the viewing device oranother display device navigation instructions to the selected one ofthe one or more locations, the navigation instructions being provided inan augmented reality such that the navigation instructions are providedin the field of view of the occupant and include directions for drivingthe vehicle to the selected one of the one or more locations, the fieldof view including an environment external to the vehicle; and at leastone i) display an item for the occupant to select on the viewing devicecorresponding to starting a game and, based on the location of the eyeand the location of the body part, start the game, and ii) determine abusiness where the vehicle is stopped and take a survey of the occupantvia the viewing device for the business where the vehicle is stopped.18. The method of claim 17, further comprising: tracking an iris of theeye of the occupant; and based on the location of the iris and thelocation of the body part, displaying the items via the viewing devicein the field of view of the occupant.
 19. The method of claim 17,further comprising: receiving information and preferences of theoccupant; and based on the information and preferences, displayinginformation about certain ones of the objects in the field of view ofthe occupant, displaying targeted advertisements related to certain onesof the objects in the field of view of the occupant, and displayingtargeted coupons related to certain ones of the objects in the field ofview of the occupant.
 20. An environment interactive system for avehicle, the environment interactive system comprising: a first cameraconfigured to capture first images of an area external to the vehicle; asecond camera configured to capture second images of an area interior tothe vehicle; a telematics control module configured to determine ageographical location of the vehicle; a viewing device configured todisplay items (i) at least one of adjacent to, on or in front ofdetected objects, which are external to the vehicle, and (ii) in a fieldof view of an occupant, wherein the items include informationcorresponding to the detected objects; and an infotainment moduleconfigured to detect the objects based on the first images and thegeographical location of the vehicle, based on the second images, trackat least one of a location of an eye of the occupant or a location of abody part of the occupant other than the eye, based on the location ofthe eye and the location of the body part, display the items via theviewing device in the field of view of the occupant, determine abusiness where the vehicle is stopped, and take a survey of the occupantvia the viewing device for the business where the vehicle is stopped.